Medical education assessment: a brief overview of concepts in generalizability theory

article distributed under the terms of the Creative Commons Attribution License which permits unrestricted use of work provided the original work is properly cited. General Medical Council (GMC) in the UK has emphasized the importance of internal consistency for students' assessment scores in medical education. 1 Typically Cronbach's alpha is reported by medical educators as an index of internal consistency. Medical educators mark assessment questions and then estimate statistics that quantify the consistency (and, if possible, the accuracy and appropriate-ness) of the assessment scores in order to improve subsequent assessments. The basic reason for doing so is the recognition that student marks are affected by various types of errors of measurement which always exist in student marks, and which reduce the accuracy of measurement. The magnitude of measurement errors is incorporated in the concept of reliability of test scores, where reliability itself quantifies the consistency of scores over replications of a measurement procedure. Therefore, medical educators need to identify and estimate sources of measurement error in order to improve students' assessment. Under the Classical Test Theory (CTT) model, the stu-dent's true score is the sum of the student's observed score and a single undifferentiated error term. Using this model, the most frequently reported estimate of reliability is Cronbach's alpha. Almost always, however, when alpha is reported, it incorporates errors associated with sampling of items, only. Accordingly, alpha does not allow us to pinpoint , isolate, and estimate the impact of different sources of measurement error associated with observed student marks. An extension of CTT called " G (Generalizability) theory " enables us to differentiate the multiple, potential sources of measurement error called " facets " (sometimes called " dimensions " in experimental design literature). For example , in an OSCE exam, a student might be observed by one of a large sample of examiners, for one of a large sample of standardized patients (SPs), and for one of a large sample of cases. The facets, then, would be examiners, SPs and cases-each of which serves as a potential source of measurement error. The set of all facets constitutes the universe of admissible observations (UAO) in the terminology of G theory. As another example, suppose that for a cardiology exam, the investigator is interested in an item facet, only; in that case, there is only one facet. There is no right answer to the question of which facets, or how many …